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METHOD OF OBTAINING AN IMAGE 



The present invention relates to a method of obtaining an image and, 
more particularly, a method of obtaining a substantially linear representation of 
5 the brightness of an image having a wide dynamic range. 

Digital electronic cameras and other similar imaging devices and systems 
often use CCD (charge-coupled device) sensors, and work by converting incident 
photons into charge and accumulating the charge in each pixel for the duration 
of the exposure. Like all other imaging devices, including conventional wet 

10 photography and other digital systems, such imaging devices impose limits on 
the dynamic range of the signal they can capture. In particular, image details in 
the regions which are either too dark or too bright cannot be captured. In such 
digital devices, the charge in each cell on to which part of an image is imposed, 
is read out and converted into a number representing its intensity or brightness 

15 via an analogue-to-digital converter. Therefore, if the image is too bright, or the 
exposure time is too long, then saturation occurs and the brightness of the image 
is clipped to the maximum representable intensity and no detailed information is 
available in these areas. This imposes an upper limit on the detected brightness. 
Conversely, if the image is too dark any signal is indistinguishable above the 

20 quantisation noise in the analogue to digital converter and therefore detail is lost 
in these dark areas. 

An important characteristic of this clipping process is that it is destructive 
and destroys information which cannot subsequently be recovered from the 
recorded image. Therefore any attempt to capture the information that would 

25 otherwise be missing, must be made at the time of capture of the image. The 
resulting image requires a wider dynamic range than that of the basic sensor and 
analogue-to-digital converter. It is possible, then, to correct overall over- or 
under-exposure by adjusting the time over which the sensor is accumulating 
charge so that the full dynamic range of the digital imaging system is utilised. 

30 However, whilst this uses the available dynamic range to best effect, it cannot 
assist where there are both very bright and very dark regions, for example, in an 
electrophoresis gel, in the same field of view as both lengthening and reducing 
the exposure time cannot be carried out simultaneously. In these circumstances 
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it is possible to capture either the bright areas or the dark areas accurately, but 
not both simultaneously. 

Now, it is evident that, under such circumstances, a series of images 
captured using different exposure times will contain all of the available 
5 information. Indeed a technique well known in the art involves simply averaging 
together a series of images recorded with steadily increasing exposure times. 
This can be shown to result in an image which is approximately proportional to 
the logarithm of the intensity. Such a result may be pleasing to the eye in that 
very bright regions do not suppress the detail of very dark regions, but such 

10 images lack the crucial linear quality often required for quantitative analysis. 

It is possible to obtain camera systems having an inherent high dynamic 
range and whose output can be digitised to a 1 6-bit resolution. Such systems can 
achieve the dynamic range necessary to image both very dark and very light 
areas without saturation, but they are both very expensive and generally slow to 

15 operate with regard to focussing and adjusting the field of view. The advantages 
of the present invention are that an inexpensive sensor and digitizer may be 
used, and the readout can be rapid, facilitating convenient, fluid adjustment of 
focus. 

The ideal digital output v' xy from an analogue-to-digital converter of a true 
20 image of intensity i xy is given by 

v; v = /m, v + c 

where T is the exposure time, K is the overall gain of the system, and C is an 
offset. However, due to the saturation at the high and low ends of the range the 
25 true output is constrained by: 

KTi xy +C v min <v; v <v max 
v xy = j v max when v' xy > v max 

V V < V 

v mm jtv mm 
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In order to reduce the effects of noise an average over a series of images with 
different exposure times T n can be taken, giving a superior estimate of the true 
image i^: 
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Averaging of this kind is well known in the art but it does not resolve the 
problem of saturated areas and dark areas and therefore the present invention 
discards the values for these pixels from the average, such that only pixels 
having significant values, that is those well away from the dark and light limits, 
are considered when averaging the images together. 

The present invention is aimed at overcoming the shortcomings of the 
prior art methods. 

According to the present invention there is provided a method of creating 
an image which includes the steps of: 

obtaining a substantially linear representation of the brightness of an 
image, the method comprising, for each of a set of pixels (x f y) in a two 
dimensional array, calculating an estimate of the true image intensity ( ^ v ) as a 

weighted average of n samples of the apparent image intensity (v ) as 




where v w 1V is the apparent intensity measured, T n is the exposure time, K is the 
gain of the system, C is an offset and w n xy is a weighting factor which is defined 

to maximise the signal-to-noise ratio and discard insignificant, that is saturated 
or near zero, values; 

A 

thereafter saving each of the values i together with other data 

representing the image; and 

outputting the image to a display or to a printing device or to a subsequent 
analysis. 

For example, in the most simplistic case: 
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A further example is that of Gaussian noise which is minimised, and hence the 
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signal-to-noise ratio is maximised, when ' = 1 and therefore 
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The invention provides for an image to be formed over a wider dynamic 
range than that of the basic sensor and A-to-D converter and enables the image 
to be a linear representation of the brightness of the original scene whilst taking 
into account all the available data and with an optimal signal-to-noise ratio. It 
also allows an image series to be captured automatically to cover a wide range 
of exposure times as the number of saturated and zero output pixels is a natural 
output from each step, so that the sequence can be chosen to cover the entire 
dynamic range of the specimen automatically rather than arriving, by iteration, at 
the optimal single exposure time for the whole specimen which is the technique 
currently employed in the art. 

The choice of the exposure times T n is very important. For practical 
cameras, the exposure time is subject to limits and not all exposure times are 
possible within those limits. As part of the implementation of the present 
invention, it is required to determine those pixels which are saturated. The 
operation of a system incorporating the invention can be broken down into three 
phases: adjustment, capture and analysis. By counting those pixels which are 
saturated and those for which there is a zero output for a particular frame, the 
implementation can determine whether the frame is generally over- or under- 
exposed. During the adjustment phase, the "best" single exposure time for the 



specimen is selected. This selection process is either performed manually via 
viewing unmodified digital image directly on screen or automatically from the 
frequency histogram of the intensity levels of the image. 

As the exposure time is varied, the number of pixels at the limits may be 
monitored. If the exposure time is reduced to a point where no pixels are 
saturated then there is no more information to be obtained by any further 
reduction in exposure time. Similarly, if the exposure time is increased to the 
point where most pixels are saturated there is little point in any further increase. 
In the presence of noise, it may be worth going a little beyond each of these limits 
so as to increase the number of valid samples for the averaging. 

This approach can be subject to two problems: sometimes the ratio 
between the successive exposure times is not precisely known and there is often 
an offset present in the camera electronics or in the analogue to digital converter 
which results in a further offset to each image. 

In order to resolve these difficulties a regression calculation can be 
performed between successive pairs of images in order to determine the linear 
function relating them. Given this relationship, each image can be transformed 
to match the scale and offset of the other. For unsaturated pixels there is a linear 
relationship between the images recorded at different exposure times, where the 
gradient is the ratio of the two exposure times and the offset is a constant for all 
pixels. 

For example if one considers: 




and 




KTj v + C 



these can be re-arranged as: 




which is of the linear form: 



V' 



r = flV V* + * 



So, for unsaturated pixels, there is a linear relationship between the images 
recorded with different exposure times, where the gradient a is the ratio of the 
two exposure times and the offset b is a constant for all pixels. The situation is 
slightly complicated by the presence of noise in both the v m and v„ images and 

5 the best fit linear relationship in this case is given by a "perpendicular" regression, 
which minimises the sum of the squares of the perpendicular distance between 
a point formed by the coordinates ( v nJ , v n2 tXy ) and the fitted linear relationship. 

An example of an implementation of a standard perpendicular regression 
technique that accounts for the possibility of saturated pixels is the following 
10 pseudocode: 

s = sL = sn = sm = snn = smm =snm = 0 

for all y values of image 

{ 

1 5 for all x values of image 

{ 

(vn is intensity value of image n at x.y and vm is intensity value of image m at x,y) 

if vn > vmin AND vn < vmax AN D vm > vmin AND vm < vmax 

{ 

20 S = S + 1 

sn = sn + vn 

snn = snn + (vn*vn) 

sm = sm + vm 

smm = smm + (vm*vm) 
25 snm = snm (vn*vm) 

} 

else 
{ 



sL = sL + 1 



30 } 
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sdndndmdm = snn - (sn * sn / s) - snm + (sm * sm / s) 
sdndm = snm - (sn * sm / s) 
aa = sdndndmdm / sdndm 



a = (-aa + SquareRoot(aa * aa + 4)) / 2 
b = (sm - a * sn) / s 
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In this way the linear relationship between one image and another may be 
determined. This is best done between one image and the next in the series, 
because, in this way, the number of spatial locations (x,y) whose intensity is not 
saturated in either image, is maximised, resulting in the best statistics and the 
most accurate estimate of a and b. 

As a by-product, sL is the number of pixels which is saturated in at least 
one of the images. 

Having determined a and b, each image can be transformed to match the 
scale and offset of the first in the series. The expression for the averaging kernel 
is then: 
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where a n and b n are the gradient a and offset b measured between image n and 
image n-1 (a 1 = 1; b 1 = 0), where: 



30 



W n rv = < 



n mm n ,.ry max 

0 when v n rv > v mjlx 

n . x v max 



0 



v < v 

n .xv min 



One example of a system constructed in accordance with the present 
invention will now be described with reference to the accompanying drawing, in 
which: 
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Figure 1 is a general view of the apparatus. 

The system as implemented in the example includes a camera 1 (in the 
particular example, a JVC KY-F50E camera). The camera 1 output signals are 
digitized by analogue to digital converters which form part of a framestore board 
2 ( a Synoptics Prysm framestore) fitted within a personal computer 3 (a Gateway 
2000 PC with a 500 MHz Pentium II processor and 256 MB RAM running 
Windows 98) and then the digital image 4 is placed in the computer's memory 
and may be displayed on a monitor 5. 

A 

Thereafter the brightness values i xy for each pixel are calculated in 

accordance with the expressions above. 

Control of the system was achieved using software written in Visual Basic 
and C++ using Synoptics Image Objects to handle the display and processing 
functions. 

In operation, firstly, during the adjustment phase the "best" single 
exposure time for the specimen object is selected. This selection process can 
either be performed via the system displaying on screen the unmodified digital 
image direct from the camera 1 and allowing the user to select an exposure time 
which is reasonably rapid yet long enough that sufficient of the contrast in the 
specimen can be seen or the exposure time may be determined automatically 
from the frequency histogram of the intensity levels of the image. 

Thereafter, during the capture phase a number of steps are under taken 
according to the present invention. First, areas of computer memory are located 
for the values needed during the process corresponding to: 



w 



n,xy 



and 



The data in these memory areas is stored in high precision format in order 
to avoid data degradation due to rounding or truncation. 

The exposure time selected during the adjustment phase is then used as 
a starting point. An image is acquired using this exposure time. For this image, 
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a is taken as 1 and b as zero, w is 1 for those pixels having an "in range" value 
and zero otherwise, and the summations identified above are initialised with the 
values indicated. The exposure time is then increased by approximately a factor 
of 2, and another image acquired. 

For each new image the following processes are carried out: 

• the image is compared to the previous one acquired to determine 
the linear relationship between the gradient a and offset b, considering only those 
spacial pixel locations in which the intensities in both images are well away from 
the limits v mjn and v max ; 

• the product of all the a values, and a sum of all the b values, is 
computed; 

• for each pixel location in which the new image of intensity value is 
within range, the values in the two formations given above are updated; (w is 
equal to the product of all the a values for those pixels having an "in range" value, 
and zero otherwise) 

• if the number of spatial locations for which the new image's 
intensity value exceeds v max is greater than a predetermined percentage of all 
spatial locations in the image, then there is little point in increasing the exposure 
time further. 

This process can be repeated to resume for the same starting exposure 
time but decrease in the exposure time for each iteration until more than a given 
percentage of the spatial locations have an intensity less than v mjn . Finally, the 
ratio of the two accumulation values is calculated to form a result image. This 
image may be subject to a linear rescaling or offset to make it suitable for 
conversion to a fixed point format image for further processing. 
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CLAIMS 



1 . A method of creating an image which includes the steps of: 

obtaining a substantially linear representation of the brightness of an 
image, the method comprising, for each of a set of pixels (x, y) in a two 
dimensional array, calculating an estimate of the true image intensity (i ) as a 

weighted average of n samples of the apparent image intensity (v fl ^ v ) as 
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(v -C 



I XV = 



where v is the apparent Intensity measured, T n is the exposure time, K is the 



gain of the system, C is an offset and w n xy is a weighting factor which is defined 

15 to maximise the signal-to-noise ratio and discard insignificant, that is saturated 
or near zero, values; 



thereafter saving each of the values i together with other data 



20 



25 



30 



representing the image; and 

outputting the image to a display or to a printing device. 

2. A method according to claim 1 , wherein a linear relationship is established 
between images recorded with different exposure times by the use of a 
perpendicular regression technique whereby each image is transformed to match 
the scale and offset of the first in the series and whereby the weighted average 
is calculated as: 



A 



2r W n,xy 



10 



where a n and b n are the gradient a and offset b measured between image n and 
image n-1 (a 1 = 1; b 1 = 0) when 



0 when 
0 



V < V < V 
mm r n,xv max 

V > V 
n.jrv — max 
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METHOD OF OBTAINING AN IMAGE 



ABSTRACT 

5 A method of creating an image 4 obtained from say a camera 1 to obtain 

a substantially linear representation of the brightness of the image includes, for 
each of a set of pixels (x, y) in a two dimensional array, calculating, in a computer 
3, an estimate of the true image intensity {i xy ) as a weighted average of n 

samples of the apparent image intensity (v„ ). This is calculated as: 

10 

I 

* _ _n 

ft n 

1 5 where v n xy is the apparent intensity measured, T n is the exposure time, K is the 
gain of the system, C is an offset and w n xy is a weighting factor which is defined 
to maximise the signal-to-noise ratio and discard insignificant, that is saturated 

A 

or near zero, values. Thereafter each of the values i xy is saved together with 

other data representing the image 4, before the image is output to a display 5 or 
20 to a printing device. 



r n H kt k ) i 4v n H t. ) 
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Computer 3 



